-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
plugin: Added filter feature per record. #42
Conversation
fluentd cannot apply filter per record currenty. This patch provides user-defined filter feature per record, even when using MultiEventStream. Signed-off-by: UEMURA Yuichi <y.uemura@ntt.com>
I think it was tough work to clear legal procedures to make your patch open:-) I understood the filtering/validating feature at the input-side is important. You can do same thing by extending |
Unfortunately, as an OSS product, it's hard to accept the patch which includes company's copyright. That could be a problem, when we change the license or donate the code to somewhere. Please reconsider it. Sorry about this. Thanks - K |
@frsyuki Yes, TailInput is enough to filter when tailing. Our problem is that some input plugins of fluentd cannot restrict the size of input. module Fluent
class ForwardInput < Input
# ...elision...
def on_message(msg)
# ...elision...
elsif entries.class == Array
# Forward
es = MultiEventStream.new
entries.each {|e|
time = e[0].to_i
time = (now ||= Engine.now) if time == 0
record = e[1]
# We'd like to hook record and to reduce output traffic.
es.add(time, record)
}
Engine.emit_stream(tag, es)
...elision...
end Input-time restriction is useful when the the owners of fluentd's instances are different. I think my filtering patch is one solution, but I don't know whether my idea is the best or not. |
@kzk Yeah, Okey, I'll delete it next patch. Thank you for your pointing out :-) |
@frsyuki Note that this patch is just a prototype... the goal of this patch is to establish the generic method to limit input size. |
So, you can't trust the sender of logs at all, right? I imagine you need "multi-tenant" capability. To satisfy your needs, the input plugin will need following features: In my opinion, a) should be done at the boundary of the "untrusted zone" and "trusted zone" not to allow unauthorized users to consume resources of the "trusted zone". And b) should be done at the boundary of the "frontend system" and "backend system". Because these validations tend to be complex and slow task because it needs be strict to prevent potential security holes. in/out_forward plugins and Fluentd's emit/write mechanisms are optimized for availability and performance. I think it's difficult to add validation features without degrading these benefits. |
Yes, you're right.
It seems like the best solution to layer between trusted and untrusted zone.
This is our next work to come true layering :-) |
fluentd cannot apply filter per record currenty.
This patch provides user-defined filter feature per record,
even when using MultiEventStream.
This problem is one proposal for solving the issue #39(https://github.com/fluent/fluentd/issues/39).
Signed-off-by: UEMURA Yuichi y.uemura@ntt.com